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ABSTRACT 
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by observation, interviews, and questionnaires. A cross-case 
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inservice events was observed and comprehensive process notes were 
recorded. A summary of the 10 case studies describes planning and 
preparation^ fYfnt actiyit P?^* ^?iP ailt evaluation , intended use 
of ideas acquired at the event, and actual use. A description of the 
scope and activities of the events includes data oh the audience, 
length of time of activities, workshop si tes , presentation methods, 
participant evaluation, and intended use of ideas • Interviews 
conducted about six months after each event provide information bh 
local activities resulting from the state-sponsored iriservice> 
actions taken, reasons for action or inaction, satisfaction as to the 
usefulness of _ the events, and individual and organizational impact of 
the event. A list of 18 coticlus ibris is presented with recommendations 
for coordinators of state-sponsored inservice. (JD) 
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Introduction 

State Education Agencies (SEAs) conduct or sponsor various kinds of 
training for local educators. Events range from lar^e conferences to small 
group meetings. Federal, state, and/or local piricfitieK are addressed, 
sometimes stimulated by a "client need" aud som, t -3£s by a "program initia- 
tive. " Ail events require considerable resources — fur. rig; time, expertise. 
A key question is: What kinds and levels of impact are made by the SEA on 
participants of state-sponsored inservice? The objectives of this 7 iper are 
to describe the various kinds of state-sponsored inservice and to explore ^he 
factors influencing impact on participants in terms of upe of knowledge u 
skills presented. This pilot study focuses on a single state and representa- 
tive training events sponsored by that SEA. Specific data were reported in 10 
separate case studies. Here, data are summarized across these cases. The 
following sections are presented: methods and measures of the study, 
findings, summary and conclusions . 

ires 



This pilot study was conducted by staff of Research for Better Schools 
(RBS) during 1983. It was designed to explore the issue of increased 
participant knowledge and/or skills resulting from inservice events sponsored 
by the Maryland State Department of Education (MS BE) , with an eye towards 
determining heeded improvements and implications for policy decisions about 
resource allbcat ions . 



* Resources available to RBS for this work were minimal. Therefore, a full- 
scale study could not be conducted. It was agreed that the measures, and. . 
methods of a small pilot study might be used subsequently (possibly by IDP 
members), if they yielded data worth exploring further. 
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the ten inservice events were selected from those offered by eight MSDE 
offices and divisions. Each of the MSDE Inservice Development Panel (IDP) 
members representing one of the eight divisions was asked to select (with 
approval from respective supervisors) two or three different kinds cf 
inservice events sponsored by that division. From the list of events offered 
by IDP members, RBS identified the ten events to be included in the pilot 
study, including all types of activities (major conference, regional workshop, 
follow-up assistance). The selected events were all conducted during the five 
month period between March and July, 1983. 

A preliminary research design was developed by RBS in cooperation with 
the IDP. Built into the design was a review process which occurred at each 
stage of the pilot study.. RBS met with staff oi: the Staff Development Branch 
and/or IDP members to review measures and methods to be used in each new stage 
of the study. A tracer analysis approach was used to determine how planning 
occurred for each inservice event, and what results there were in the months 
which followed. Also, each event was observed by RBS, and participant evalua- 
tion results were analyzed. 
Data Collection 



Three general methods of data collection were used: observations, inter- 
views, and questionnaires. 

• observations . Process observations of each of the ten 
critical inservice events were conducted by RBS staff. For 
each of the events, comprehensive notes were taken, objec- 
tively describing what occurred and indicating time elapsed. 

# interviews ♦ Telephone interviews were conducted by RBS with 
the MSDE event coordinators, LEA planners, and presenters, as 
appropriate. These interviews focused on the nature and 
extent of planning for the inservice event. 

« Questionnaires > During the course of the study three different 
questionnaires/survey instruments were used. They included__ari 
Evaluation Fornu Intended Use Survey and Inservice Fol low-Up 
Surveys 
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- The Evaluation Form was developed by MSDE as the standard forth 
to be u^ad in the evaluation of scheduled inservice activi- 
ties.. In. some cases slight modifications of the form were 
made by MSDE event .coordinators to provide more relevant 
information. The form was to be completed by all participants 
of each event. I tenia on the form related to clarity of 
workshop objectives , quality of content and materials, 
effectiveness of p esenter(s) , and overall effectiveness of 
the workshop. MSDE event coordinators took responsibility for 
tabulating participants' evaluations. 

- The Intended Use Survey was developed by RBS to be completed 
by at least one representative of each agency (LEA, institute 
of higher education, public library) attending the inservice 
activity. The survey concentrated on participants 1 intentions 
regarding the use of information, ideas, materials , and skills 
obtained from the inservice activity they had attended. In 
addition, the survey asked participants to indicate their 
willingness *".o provide additional information later about 
their application of ideas acquired at the inservice event; 



- The Inservice Fol low-Up Survey, designed by RBS, was to be 
completed by a sub-sample of participants who had indicated 
interest (on the Intended Use Survey) in providing RBS with 
additional information on their actual use of the information 
and materials obtained from the inservice activity. The 
survey consisted of a series of telephone interviews , and a 
survey questionnaire inviting responses from thirteen selected 
participants. The sample of respondents to this survey was 
selected in the following way: for each event a single 
participant was selected to provide follow-up datai T ^ ree 
exceptions occurred — three of the ten events (attended by 
over 45 participants) provided two respondents each. This 
res lted in thirteen potential respondents • Respondents were 
chosen so that a large urban LEA, medium and small rural LEAs, 
colleges, and a public library would be represented. 

Data Analysis and Reporting 

The data were initially analyzed separately for each of tne ten critical 

events. Preliminary case reports on each event were written, and included 

sections describing the actual event * prior planning and preparation; parti::! 

pants 1 evaluations; participants' intended use of the ideas presented; and 

factors influencing the success or failure of the event. These case reports, 

which did not yet include sections on local follow-up use, were sent out in 

draft form to the relevant MSDE event coordinator, with a request to inform 
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RRS of any inaccuracies or inadequacies in the report. When all corrections 
to this portion of the case report had been made* a copy was sent to the chair 
of the IDP (Coordinator, Tnservice Profeusibrir. Development:), and tho relevant 
MSDE Division Director was notified of the report's availability (Summer 1983). 

The data on local follow-up use were collected, analyzed, and sent to the 
event coordinators as the final sections of each of the case reports (February 
1984); The ten case reports were developed for the use of the Division staff 
involved in inservice* This report represents a cross-case synthesis of the 
ten individual case studies, and is intended for a vider audience. 

Findings 

This section summarizes the results of the ten case studies. It 
describes planning and preparation, event activitv.es, participant evaluation 
and intended use of ideas acquired at the event, and actual use. 
Planning and Preparation of Iriservice Events 

As stated earlier, the ten events varied in several ways. Table 1 
presents a summary of the key characteristics. Of interest is the fact that 
five events were pre-contracted, meaning that participants agreed prior to the 
event that they vould carry but specific activities related to MSDE input. 
Since some events we^re follow-ups on previous activities, and others wer ? new 
initiatives, the nature and extent of planning varied among events. Aspects 
of planning examined included: participation, use of time, selection of 
content, selection of presenters, and design of delivery processes. 

Participation in Planning . Planning for the ten inservice events was 
accomplished in a variety of ways by different numbers of people from various 
groups or organizations. For instance, one event was planned and carried out 
by a single MSDE staff person with just a little consultation with intended 
participants. In another case, two MSDE staff undertook planning. In contrast, 




Table 1 

Key Characteristics of the Ten Inservice Events 



Division 


Subject/Content 


Type of Event 


Approx. No. of 
Participants* 


i 


~ — — — — — — — — ■ 

Food Services 


o ta tewiue LonieiciiLc 




2 a 


Teacher Effectiveness 


Single LEA Follow-up 
(pre-contract) 


21 


2b. 


Research m readier Education 


Cf oh gt:H /4 o Pori f OT*ori r P 


60 


3 


Teacher Effectiveness 


Statewide Follow-up 
vpre-contrdct y 


6i 

21 


/" 
4 


Functional Mathematics 


Regional Workshop 


5 


Correlation Workshop 


Single LEA Workshop 
(pre-contract) 


6 


na 


instructional Resource Network 
• 


Single LEA Workshop 
(pre-contract) 


26 




Long Range Planning 


Regional Workshop 
(pre-contract) 


4 




v r a fO'/i t ionul Programming 


Statewide Conference 


31 




i 

?li : 'j-ioirp titers 

i 


Statewide Conference 


65 



'•'uaibV-.s are based on estimates of on-site ob serve ts; 
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two events were planned by large committees — one Including represi>iirHtiv»s 
from three MSDE divisions, the other including MSDE staff and representatives 
from eight LEAs . 

It appeared to be most desirable for representatives of event participants 
to be directly involved in planning. It was almost as valuable if participants' 
needs and preferences were taken into account by MSDE coordinators who clearly 
understood those needs (e.g., through a survey or long-term interaction). 
Ideally all participants — MSDE, presenters, system members — had (prior to 
the event) a clear and common set of expectations about the content, its 
relationship to their work, and "iow they could or would use it. Insufficient 
awareness of participant expectations or of their relative expertise caused 
participant frustration and low application of ideas presented. 

Planning Time . Time invested by coordinators and tie length of time over 
which planning occurred varied. 'Planning for some events began two or three 
months before the activity was held, but in other cases planning began two 
years before the actual event. 

Planning time for inservice activities ranged from one full day including 
phoning and gathering of materials plus a meeting with LEA staff, t;o fifty 
days of a coordinator's time which included time spent not only in preparing 
for the presentation, but all work related to the project for which the 
inservice was conducted. 

Planning for each of the events generally involved meetings with other 
staff, p' ione conversations t arid gathering arid organizing of materials . For 
two conferences a significant portion of the time was needed to preview 
mate-rials to be used at the events. 



Total time spent on planning (by a coordinator arid/or teams) was greatest 
when coordinators wanted or needed to develop ex-vSrtise in the subject of the 
event and/or to develop materials to be used with participants. Planning time. 
.J3S fairly high when the event was complex In terms of the number/ type of 
objectives and subjects addressed. Complexity sometimes related to the 
duration of an event (but not always). Least planing time was spent when 
planners had a good understanding of the content , and vere capable process 
designers. The amount of time spent had less bearing on the quality of the 
event or its impact than did how the time was spent. 

Selection of content /presenters /delivery process . Theoret ically the 
selection of content , presenters, and delivery process should be determined by 
the objectives set for each event. In addition, events having objectives of 
different levels (e.g. , awareness, skill building) of the same subject should 
be easier to design than itl cases where several subjects are^addressed * 

Table 2 presents the number of subject areas .covered and objectives set 
for each of the ten critical events. As indicated in the cable, there were a 
total of thirteen subjects and thirty-three different objectives. Nine of the 
ten events presented material on a single subject; one event presented 
material on four different subjects. Three of the events set only one 
objective each, while the remaining seven activities set multiple objectives 



ranging from three to six objectives each* 

Tabic 2 

tfumber o. Subjects and Objectives Addressed by t!.e 1 :n Events 



SSSss v Division 
























^S- sponsor 


. 1 


2a 


2b 


3 


h 


5 


Ga 


6b 


7 


8 




Subjects 
























& ^Vr 

v% objectives ^^Sc 























To t :i 1 


Number of Subjects^ 


4 


I 


1 


I 


- " 
i 


I 


I 


1 


1 


1 


n 


Number of Objectives 




I 


3 


U 


u 


1 


6 


i 


5 

._ 


5 


13 
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Since the planning process is influenced by the types of objectives set 
for an ihservice activity, objectives .were analyzed according to their rela- 
tionship to the Bruce Joyce Inservice Training Model. Joyce's model basicall 
states that if the purpose is for participants to use the new material 
presented to them, the ihservice activity must be structured to include 
certain training components, (The activity may be a single event, but, in 
Joyce's model, is usually several events spread over a period of time, 
allowing participants to "digest" new learning and practice in their home 
environment . ) The four training components are: Rationale & Theory Building 
Demonstration/Modeling, Practice /Feedback and On-site Coaching. Bruce argues 
that the higher the level of the component, the more likely it is that many 
participants will apply ideas presented. He states that, in general, 
"rationale and theory building" results in \awareness; "demonstration/modeling' 
leads to conceptualization; "practice/feedback" results in skill development 
through application; and "on-site coaching" leads to successful application* 
This framework can (theoretically) be used tc examine activities planned in 
terms of objectives (outcomes) stated. 

Table 3 categorizes each of the ten events (as new initiative, follow-up , 
precontracted) and provides a count of the number of objectives which fall 
into each of the Joyce training component categories. In total , eighteen of 
the activity obj ectives fell into ths "rationale and theory building" category 
twelve into the "practice/feedback" category , five in the "demonstration/ 
moueling" category and only one in the "on-site coaching" category. With one 
exception, all seven "new initiative" events included rationale and theory 
building. (The exception was pre-contracted and designed as a "hands-on 1 ' 
workshop for a small number of participants . ) All follow-ups a] so included 
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-Table 3 



Categorization of Event Objectives; According to Bruce Joyce Training Components 



* Event classifications: A = new initiative C = pre-contract 

B = follow-up 



\r Event* 


l 


2 


2a 


3 


I 


5 


uU 




7 

* 


c 




Training \ 
Components \ 


A 


BC 


A 


Be 


A 


AC 


AC 


BC 


A 


A 


Total** 


Rationale and 












i 












Theory Building 


1 


1 


: 1 


2 


2 




2 


1 




5 


18 


Demonstration/ 
Modeling 

0 






1 


1 






2 


. 1 


/' 

/ 




5 


rracLice/ 


- 






















Feedback 


3 




1 


2 


3 




2 




-1 


I 


12 


On-site Coaching 












i 






i 




1, 


Actual Number 
























of Objectives 


4 


i 


3 


4 


4 


1 


6 


1 


4 


5 


33 ' 



























Cote: Some objectives as written fell into .more than one category, therefore, for three events the 
actual number of objectives stated is less than the number of components addressed. 
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rationale and theory building, two inclt 5d demons t ration/ modeling, arid only 
one included practice/feedback, (Yet, according to the Joyce model, one might 
expect greater emphasis on the latter objectives for pre-contracted 
follow-ups . ) 

The ways in which the content, presenter, and delivery process were deter- 
mined varied for each of the inservice events. Iii addition to the influence 

of objectives, the content and topics were also determined in one or more of 

it - - - 

the following ways: \ 

v 

- based on demonstrated need as perceived by MSDE Staff (5)* 

- by the planning committee (4) 

- resulting from comments on the standard MSDE Evaluation Form of a 
previous activity (2) 

- based oh results from a formal needs assessment (1) 

- based on an assumed need possibly related to a previous inservice 
activity (1) 

The selection of presenters also came about in a variety of ways. These 
are listed below: ' 

- planning committee (4)* 

- expert in the field (especially in the case of MSDE coordinators 
conducting their own activities; coordinators were 1 most intimately 
aware of the needs of their audiences) (3) 

- experience with and knowledge of speaker's capability (2) 

- reputation of speaker (2) 

- volunteers (1) 

The delivery process for the inservice activity was determined in the 
following ways: 



- past experience of a successful delivery process (4)* 

- planning committee (3) 

- knowledge of speakers^ presentational styles (1) 
~ data from MSDE Evaluation Form (1) 

- set up to model the process participants are expected to learn (1) 



* Numbers represent a count of the number of events using a particular method. 
More than one method may have been used for a single event. 
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The selection of content, presenter, and delivery process was not 
necessarily influenced Oy a single factor; In some cases a series of factors 



contributed to the final decision. For example, the selection of a keynote 
speaker for one conference was based on the speaker's national reputation as 
well as MSDE staff's knowledge of her capability. The presenter was then 
resporiible for designing the process. 

Follow-up. Follow-up activities (with two exceptions) were generally 
planned during the initial planning phase for each of the critical events. 
In the seven cases where follow-up was planned, activities included: 

- dissemination of videotapes (1) 

- technical assistance as necessary (2) 

- MSDE to initiate discussions about new program development (1) 

- regarded as largely the responsibility of the LEA (1) 

- participants to be brought together again for technical assistance 
and inservice offered by participant groups (1) 

- regional workshops (i) 

- local inservice workshops to initiate discussions on how to implement 
and develop action plan (I) 

In some cases, follow-up was planned, but specific activities were not deter- 



mined during the planning phases 

Summary . Analysis of the available data on planning of the events 
indicated that a number of factors influenced the subsequent success of the 
inservice activities. Certain factors in planning facilitated the accomplish- 
ment of objectives. Other factors created barriers. 

Factors related to planning that contributed to the successful accom- 
plishment of event objectives included: 

• frequent contact by MSDE to work with and prepare presenters 

• presenter's famili arity with local activities and needs, and 
positive working relationship with participants 



* Numbers represent a count of the number of events using a particular method. 
Mo ire than one method may have been used fOr a single event. 
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• the development of Administrative Networking Teams responsible 
for developing action plans 

• timing of the workshop to take advantage of participant 
motivation/need for assistance 

• design of workshop activities that matched the objectives 
using a process appropriate to the skill bein 0 taught 

o periodic communication with LEA staff throughout the planning 
stage 

• local "ownership" of the activity 

• organization and planning by the coordinator to insure a 
smoothly run activity 

• design of workshop to include participation of several levels 
of staff 



• selection of content that could easily be applied 

• process included practice as well as presentation of informa- 
tion. 

Factors that contributed to problems included: 

• presenter was out of the country during the planning phase 

• presenter had been over-scheduled during stay which resulted 
in minor scheduling difficulties 

• participants' need for information before the presenter was 
fully prepared 

• presenter's low level of expertise in process, which included 
attention to personal concerns and insufficient attention to 
participants' interests 

• lack of common understanding among coordinator, presenter (s) , 
and/or participants of session objectives or expectations 

• poor alignment of an activity or presentation to the objec- 
tives and relative expertise or heed of participants (i.e. , 
the activity was either too simple or too advanced for the 
participants, and did riot fit the objective) 

• inappropriate design of the small group task 

• agenda as planned could not be completed in allotted time 

• facilities which could not accommodate the size of the group. 
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The influential factors that have been discussed in this section repre- 
sent a collection of all factors influencing the ten critical events. Each of 
the listed factors related to one or more events. This review of factors can 
be helpful in alerting others to potential pitfalls arid in suggesting 
strategies that contribute to success. (It should be rioted that planning is 
often intuitive, or draws or* cumulative experience: few planners are fully 
aware of the steps, aspects, or influence:* discussed above, ye.t such awareness 
would probably increase effectiveness.) 
Description of the Ten Inservice Events 

Each of the ten events was observed by RBS staff , and comprehensive 
process notes were recorded . This section summarizes the scope and activities 
of the ten inservice events . 

Audience. Table 4 summarizes the scope of the ten events in terms of 
audience, length of activity, number of presenters, and number of participants 
As indicated in Table 1, three conferences were statewide with invitations to 
all 24 LEAs in Maryland • One foiiow-up served those LEAs, from various parts 
of the states that haa pre-contracted. The Conference 2b had a statewide 
audience of college faculty. Two events were for regional audiences only (brie 
of which was for public library staff) . The remaining three activities were 
for single LEAs , all of which had pre-contracted . 

Time. The length of each of the activities ranged from a half day to 
three days . 

Location. Table 5 summarizes the different workshop sites. Three of the 
ten events were held in schools and two each in Board of Education buildings, 
hotels and University Campuses.,. An office of a Public Library was used for 
one event . Coordinators attempted to select facilities of an appropriate 
size, in a convenient location, balancing cost and comfort. 
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Table 4 

Descriptive Summary of the Ten In service Events 



' Events 










1 ' 












Characteristics 


1 


2a 


2b 


3 


4 


5 


6a 


6b 


7 


8 


Number of Invited LEAs* 


24 


1 


NA 


12 


5 


1 


1 
1 




24 


24 


Length Of Activity (days) 


2 


h 


2 


3 


1 


2 


h 


h 


1 


1 


Number of Presenters 


9 


1 


1 


2 


2 


1 


i 


1 


2 


5 


Number of Participants** 


60 


21 


60 


61 


21 


6 


26 


4 


3i 


65 



* No LEAs were invited to the Conference 2b since it was designed for, college 
faculty. Three county libraries were represented at workshop 6b. 

* Numbers presented are based on estimates of on-site observers. 
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Locations of the Ten Events 





Site of Workshop 


i 

Number of Events Using 
inis iype or bite 


School 


3 


Board of Education 


2 


Hotel 


2 


University Campus 


2 


Office in Public Library 


1 







Presentation methods . The number of presenters at each of the activities 
ranged from 1 to 9. For most activities, only one presenter was involved. 
Presentation methods and features are presented in Table 6. The lecture was 
used as a presentation method in all ten events, whereas films/videotapes were 
featured in only five of the ten events. Interactive dialogue between presenter 
and audience was a feature of all ten events with the exception of Workshop 2a. 
Workshop 6b was the only event to use all of the presentation methods and 
features listed in the table. Workshop 6a used four of the listed elements. 
This is consistent with the fact that the half-day activity was conducted to 
provide technical assistance to three local representatives. Conference 8 was 
somewhat limited in its use of different presentation methods, especially 
since it was a statewide conference with involvement of five presenters. 

Generally, workshop participants spent most of their time listening to 
presenters lecture. The second most frequent way participants spent their time 
was engaged in large and small group activities. Viewing film and videotapes 
was the next largest way in which time was spent. Finally, interactive dialogue 
between presenters and participants accounted for the smallest omount^fif time. 

IS 
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Table 6 
Methods/features of events 

























^s,- Events 


1 


2a 


2b 


3 


4 


5 


6a 


6b 


7 


8 


Method/Feature 






















iicc Lure 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Handouts 


X 






X 


X 


y. 


X 


X 


X 


X 


Overhead transparencies 


X 


X 


X 


X 


X 




X 




X 


X 


Demonstration 


X 


X 


X 


X 




X 


X 






X 


Interactive Dialogue 


X 




X 


X 


X 


X 


X 


X 


X 


X 


Testing for Understanding 


X 


X 


X 


X 


X 




X 








Activity/Practice 


X 




X 


X 


X 


X 


X 


X 


X 




Film/Video Tape 




X 


X 






X 


X 




X 
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Table 7 describes the ten events in terms of the inclusion of the Bruce 
Joyce training components as determined by the different activities comprising 
each workshop. One of the events included all four of the Joyce training 
components. All events included "rationale and theory building" components. 
Most events included "demonstration/modeling" components. Only half of the 
events included "practice/feedback" components. Two (both with less than five 
participants) included some "on-site coaching." A comparison of this table 
with Table 3, which relates workshop objectives to the Jo^e training compo- 
nents, reveals that objectives were consistent with the actual activities 
carried out. In some cases* levels of activities were conducted in addition 
to those suggested by the initial workshop objectives. 

While most participants prefer that inservice events allow for some kind 
of participatory activity (which may be categorized as "practice/feedback" or 
"coaching" or could include opportunity for question and answer sessions 
during "demonstration/modeling"), such preference does not necessarily lead to 
application. Results of this study provide some support for Joyce's findings, 
but there is strong evidence that factors other than the activity design are 
much stronger in influencing application of ideas by participants (most of 
whom are administrators) of state-sponsored inservice. 

Participant evaluation and intended use . Participants completed 
evaluation forms and brief questionnaires about intended use. Results are 
summarized here. 

Standard evaluation forms were used with the exceptions of Events //2b, 
3 and 7.* Participants rated the events using a five-point scale with 

* In each of the three expectations , some items were the same as those on 

standard forms.. Additional items were used in all cases with #3 using 

separate forms for each major session. Table 8 includes all items relating 
to the overall event. Participants of Event 6b were not asked to respond to 
evaluation questions. 
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Table 7 



Bruce Joyce Training Components Included in the Activities of the Ten Critical Events 



Event 

Training \ 
Components \ 


1 


2a 


2b 


3 


I 


i 


6a 

V 

V 


6b 


7 


8 


Rationale and 
Theory Building 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Demonstration/Modeling 


X 


X 


X 


X 


X 


X 


X 






X 


Practice/Feed Uck 


X 






X 


X 




X 




X 




On-site Coaching 












X 




X 
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S^Excellerit and l^Pobr. (See Table 8.) Overall mean ratings on the standard 
form ranged from a low of 3.98 (facilities provided foi the activity) to a 
high of 4.53 (clarity of objectives). In general, ratings were predominantly 
greater than 4.00, indicating: participants considered the content to be of 
high value and quality; materials and resources were considered good; 
presenters were viewed positively; activities and experiences were Judged as 
practical; opportunity for group participation was good; the personal applica- 
tion of the activity to participants' work was strong; the format and use of 
time during the activities was generally good; and overall, the activities 
were useful experiences for the participants. ^ There was one exception, with 
mean ratings of 3.80 or below. Participants were dissatisfied with the 
following aspects of that conference: (1) poor facilities, (2) impracticality 
of the activities/experiences, and (3) low level of need for the training as 
provided. 

As part of the evaluation of activities > respondents were asked to offer 
commepfcsr on both the positive and negative aspects, by responding to open- 
ended questions on the evaluation forms. The list below summarizes the 
positive faatures, and indicates the number of events in which a significant 
percentage of participants included the feature as highly positive. 

Presenter/presentation (7)* / 
Workshop content (3 J 

Supplemental conference materials (2) 
Workshop overall (2) 
Small group activities (2) 
Videotapes (1) 
Use of modeling (1) 
Practice opportunities (1) 
Planning/organization of workshop (1) 
Facilities/location (1) 

The following list of negative aspects summarizes suggestions for 
conference improvement made by a significant percentage of participants. 



* Number of events. 
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Table 8 



Ratings Assigned for Workshops on 
Specified Criteria for Division training 




M5DB __ L 

Evaluation 

"Criteria 



S»SB 



1, Hbw clearly were the objectives presented Co 
participants? 

2; How veil t/orc the objectives met? 

How did you rate Che value and quality of the 
content? 

4; What was the quality of the materials and 
resources? 

5, How did you rate the facilities provided 

for the activity? 

6. How did you rate the major presenter(s)? 

7, How did you rate the practicality of the 

activities/experiences? 

8. How did you rate the level of group 

participation? 
9; How did you assess; the personal application 
of the activity to your vbrk? 

A _ _ 

iC. How did you rate the fofmat/ocganization 

of the activity? 
11; How did you rate the use of tine during 

the activity? 
12. (tow did you rate the activity overall? 

a. How relevant were the objectives to your work? 

h. How did you rate the quality of ?h presentation? 

c. How did you ;ate the relevant new Knowledge? 

i How do you rate the format; use of tine: 

Tuesday?... 
Wednesday? 

c . The instructor was knowledgeable about the 
subject area of training 

f. This training was valuable to Be. 

g . What was your twl of need for training in 

this arei? 



<;«3 

1.13 

4,28 

4.29 

4.21 
4,47 

4.15 

4:24 

4.21 

4.24 

4.23 
4.33 



2a 



"21 



2b- 3 



K-60 



4.76 
4,81 

4.76 

4.57 

4.43 
4.48 

4.48 

3.43 

U? 

4.48 

4.38 
4.55 



4.80 


4.55 


4.73 


4.46 


4.63 


4.64 




4.45 




4.69 



4.93 

4;73 
4.88 
4.41 

4.59 
4.52 




Poor use of time (4)* 

Inappropriate level of information (2) 

Poor presenter (2) 

Need more hands-on activities (1) 

Need more group participation (1) 

Need small group activities (1) 

Misleading conference title (1) 

More video monitors needed (1) 

Room too cold (1) 

Room too crowded (1) 

p oor lunch and inadequate parking (1) 

in general, these negative comments (with the exception of the last three) 
reflect on the design of the event and its relationship to participant needsT 
Such negative comments were rarely made when planning incorporated such 
positive factors as those listed on pages 11 and 12. 

Intended use. Generally, one person representing each LEA in attendance 
at the training activity was asked to complete an intended use survey. 
However, in two cases all participants were asked to complete the form (staff 
of institutions of higher education, and participants of a single LEA event). 

The form listed eight ways in which participants might use information, 
ideas, materials, or skills presented during the event, and asked participants 
to check those relevant to, their intentions. (See Table 9.) For purposes of 
analysis, "Intended Uses" can be grouped into the following categories: 

(1) materials dissemination (A and B) 

(2) ideas discussion ( c and D ) 

(3) personal use (G) 

(4) staff development (E) - - 

(5) incorporation into policy or practice (F) 

These categories also represent a rank ordering of levels of use with (1) 
"materials dissemination," being the lowest level of use, and (5) "incorpora- 
tion into policy or practice," being the highest level of use. 

Table. 9 presents mean percentages of respondents' intended actions 
resulting from conference participation. Overall mean percentages indicated 
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Table 9 



Perc^of Respondents Proposing Activities 
Related to Division Training Events 



to 

to 




Give materials to immediate colleagu 

B. Duplicate/disseminate materials to other 
system employees^ 

C: Share ideas (informally with colleagu, 



D ' Sha " id f (infernally ulth other system 
employees). 

E. Conduct. inservlce/sUff development 
activities; 

" F; Corporate information into svstem 
policy/practice. 

C Use ideas/skills "on the job" myself. 
H. Other 
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that most respondents intended to: (G) use ideas/skills "on the job myself*" 
and (C) share ideas (informally with colleagues). These two intended uses 
represent moderate to low level use* Less than 50% of the respondents 
intended to: (A) give materials to immediate colleagues, or (B) duplicate/ 
disseminate materials to other system employees, or (E) conduct inservice/ 
staff development activities. Overall, greater than 50% of the respondents 
indicated intentions of the highest level of use: (F) incorporate information 
into system policy/practice. 

Summary . Participants of the ten events evaluated the inservice activi- 
ties and also indicated how they intended to use the ideas presented at the 
activities. Participant ratings for items on a standard evaluation form 
generally indicated very favorable reactions to the various aspects of the 
training events. Open-ended participant comments highlighted the presenters/ 
presentations and workshop content as features that were repeatedly positive 
across inservice events. Poor use of time, inappropriate level of information, 
and poor presenters were problems which occurred in more than one activity. 

Participants 1 indications of intended uses of the ideas presented at 
workshops suggested that most participants would use the ideas/skills them- 
selves and share ideas with colleagues. Intended use was influenced not only 
by the event objectives arid activities, but also by the type of event (initial 
statewide conferences were less likely to get widespread high levels of use 
than were pre-contracted events or regional workshops) , and by the relevance 
of the topic to a current local priority. The last factor was a very powerful 
influence arid could stimulate a high level of intended use even when MSDE 
event objectives arid activities focused primarily on rationale and theory 
building. 



28 

23 



Local Activities Resulting froth State-Sponsored Inservice 

Tn response to individual interviews (conducted about six months aJ'ter 
each event) , participants of state-spbrisbred inservice events stated what 
actions they had tp^c.n, reasons for taking (or not taking) action, and the 
impact of their actions. They also suggested ways in which MSDE might have 
helped to increase application. Interview questions and elements of 
interviewers 1 responses were used as a basis for a mail survey. Results are 
reported here. 

Actions taken . Participants reported taking between four and seven 
actions as a result of state-sponsored inservice . (See Table 10.) Informal 
discussion arid sharing workshop materials with colleagues were the most common 
actions (claimed by all respondents) . Less common actions were use of ideas 
in local inservice, and incorporation of ideas into local policy or practice 
(56% and 45% respectively) . "On the job use" (89%) usually meant that 
participants incorporated some of the knowledge gained into their everyday 
activities , for instance by being aware of additional performance criteria as 
they evaluated or assisted teachers. 

Reasons f or 1 acx^Lott^r inact±on . For those actions taken, respondents 
indicated one or more reasons for acting. For those activities not carried 
out, reasons creating barriers were also identified ; Tables 11 and 12 present 
reasons — in their relative order of power — for activities taken (or not 
taken) . 

The reasons for action most cited were the relationship of the MSDE event 
to a local priority, the usefulness of materials, and timeliness of the topic. 
By contrast, provision of on-site support from MSDE and "encouragement 11 of 
local administrators , were not as strong reasons for action . When no action 
was taken, it was most often due to scheduling problems , or lack of time and 
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Table 10 



Local Participant Actions Resulting. From 
State Sponsored Inservice (during 1983) 





Actions 


% Participants 


carrying 






out action 


(N»9) 


A. 


Gave materials to immediate colleagues 


100 




B. 


Duplicated/disseminated materials, to other system employees 


78 




C. 


Shared ideas (in|ormally with colleagues) 


100 




D. 


Shared ideas (informally with other system employees) 


89 




E. 


Conducted inservice/staf f development activities 


56 




F- 


Incorporated information into system policy/practice 


A5 




G. 


Used ideas/skills "oh the 30b" yourself 


89 
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Table ii 



Reasons for Taking Various Actions as a Result of State-Sponsored 
Inservice (during 1983) , by Percent of "Actors" per Activity Area* 







% "Actors" 


Per Activil 


i 




Reasons N= 

* 


A 
9 


B 

7 


C 
9 


D 
8 


• E 
5 


F 
4 


G 
8 




- ■ 














1. The original MSDE activity-related to a 
local priority. 


78 


57 


89 


75 


80 


100 


100 


2, Materials were extremely useful, worthwhile. 


89 




0/ 


/ J 


IDD 


100 


75 


J* lXUlCiy LUplL i-LU^Ul LallL uU UJ.oOCLU.LUa Lc t 


89 


29 


78 


50 


100 


109 


50 


4; MSDE encourage/recommended us to do this; 


67 


57 


67 


50 


60 


50 


38 


5. It filled a strong need. 


45 


43 


33 


13 


80 


75 


38 


6, My administration (superiors) encouraged/ 
supported/pushed this activity. 


56 


29 


33 


13 


60 


75 


13 


7. MSDE provided active on-site support. 


39 


14 


22 


0 


40 


50 


0 



Activities: A. Gave materials to immediate colleagues 

B. Duplicated/disseminated materials to other system em] 

C. Shared ideas (informally with colleagues) 

D. Shared ideas (informally with other system employees) 

E. Conducted inservice/staff development activities 

F. Incorporated information into system policy/practice 

G. Used ideas/skills "on the job" yourself 

Ut 3i 



ees 
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Tab, 12 



Reasons for Taking No Action as a Result of State Sponsored Inservice 
(during 1983), by Percent of "Noil-Actors" pr- Activity Area* 







% "Non-Actors" Per Act: 


tvity 






A 


B 


e 


D 


E 


F 


G 




Reasons N= 


o 


2 


0 


i 


4 


5 


1 


h 


Scheduling problem; 




50 




100 


25 


40 




2. 


Insufficient time. 




50 






50 


40 




3. 


Insufficient resources (funding, staff). 




50 






25 


40 




4. 


Not applicable to my situation. 












40 


100 


5. 


Didn't see need /importance for this activity. 




50 








20 




6. 


Not enough interest from others. 




50 












7. 


Too soon: will do as project proceeds. 










25 


20 




8. 
9. 


No opportunity. 

Lack of support from administration. 










25 


20 





Activities: A. Gave materials to immediate colleagues 

B. Duplicated/disseminated materials to other system employees 

C. Shared ideas (informally with colleagues) 

D. Shared ideas (informally with other system employees) 

E. Conducted inservice/staff development activities 

F. Incorporated information into system policy/practice 

G. Used ideas/skills "bri the job" yourself 
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other resources. Less influential reasons were lack of administrative support 
or interest from others. However, if strong reasons supporting action were 
apparent (e.g. * current local priority), reasons for inaction were usually 
overcome. 

These overall results differ somewhat for each of those actions likely to 
have greatest impact: (1) "on the job" use was most likely if the MSDE event 
related to a local priority and materials provided were worthwhile, but action 
was not taken when MSDE inservice was not applicable to the participants 1 
situation; (2) "turnkey" training by the participant was most likely if the 
topic was timely, filled a felt need, and related to a local priority; but 
reasons for not conducting local inservice ail related to insufficient time 
arid other resources; (3) incorporation into policy or practice occurred in 
less than 50% of the cases, and when it did it was because the topic was 
timely, fitted a local priority, and was supported by worthwhile materials, 
but reasons for inaction related to insufficient time and other resources, 
and/or lack of perceived need or relevance ; 

These results suggest that if the purpose of ah MSDE inservice event is 
to change individual behavior "on the job," trigger turnkey training, or 
influence local policy or practice ; attendees should be people to whom the 
topic has direct relevance, the topic should be related to a local priority, 
should be presented in a timely fashion; and should be supported by worthwhile 
materials. This suggests that pre-contracting is valuable for Jhigh level 
application. 

Expectations arid usefulness . Almost all respondents stated that the 
events they had attended had met their expectations * arid that the ideas, 
materials, general content, and /or expertise of presenters were good. 
However, about half of them suggested that MSDE could have increased local 




impact if the state department had: provided funding or other incentives for 
pilot programs, provided training for more staff (other role groups involved 
with given content area), or identified "experts 11 to assist with local 
inservice. (These suggestions appear to be strategies for overcoming the 
barrier of insufficient time or other resources. They require high MSDE 
investment and may only be cost-effective if state and local priorities match, 
and participants are willing to pre-contract.) 

Seven people responded to the questions: "What did the MSDE inservice 
event do for you? How useful (relevant to your work) was it?" One simply 
said it was useful, two gained awareness or knowledge, and four gained skills 
(of which two applied those skills directly). Of the four events in which 
skills were acquired, three were "follow-up" events. The fourth was primarily 
a knowledge-building eveht,£arid coincided with a local priority, and strongly 
felt "timely" need. Of the two events in which knowledge was acquired, one 
had been designed to bring about behavioral change (and failed for this 
respondent). The other was designed to result in application (for one role 
group) and knowledge (for another role group of which the respondent was one) . 

Individual impact . In terms of impact on individuals, all 10 events 
increased participants 1 knowledge base, with varying levels of use e.g.* 
increased clarity or specificity in a task already being done, to understanding 
of a different /better /new way of accomplishing a task. Eight events resulted 
in skill development and direct use of those skills by participants, ranging 
from modification of the way an existing task was carried out to implementation 
of a new activity or program component.* Usually, participants incorporated 
relevant information into their existing assignments, and believed that MSDE 
ha.d provided them with "important * 11 "useful." "practical" knowledge. 

* The two events that did hot develop participants 1 skills had objectives and 
activities focusing primarily on rationale and theory building. 
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Organizational impact . Respondents answered two questions: (1) "What is 
the extent of the impact this activity had, e.g., the number of people 
involved in any Activities you initiated?" and (2) "As a result of the MSDE 
event you attended, what if anything, is being done differently at the school 
or department level?" Results are summarized in Table 13, and are related to 
types and numbers of events stimulating a given outcome.* 

Collectively, for the 13 respondents, the 10 events resulted in two 
instances of system-wide change, two instances of programmatic change in one 
or two schools, and two instances of changes in teachers' behavior. Also, 
three inservice (turnkey training) sessions were conducted, and there were 
several cases of increased knowledge and improved expertise. All events had 
some kind of local impact. An analysis of objectives and types of activities 
(in terms of the Joyce components) found no significant relationship to 
outcomes; However, there was a relationship between the overall purpose of 
the project or activity of which the event was a part when the event was 
pre-contracted with participants; That is, when MSDE negotiated ahead of time 
that certain purposes would be served, certain outcomes accomplished, partici- 
pants did indeed accomplish those outcomes. The relative level (e.g., change 
in practice/policy vs. increased knowledge) was then determined in the pre- 
contracting. Quantity and quality of impact were not determined by the study, 
but there were indications of considerable variation, with local situational 



** These outcomes do not always relate to actions taken as reported by respon- 
dents. These outcomes are real — supported by observable evidence, but 
"actions reported" (or actions reported as hot taken) are less reliable • 
This is_ a function of the data collection method: participants responding 
to check lists (of actions taken, or not taken) gave more or different 
responses than were supported by their responses to open ended questions 
and interviews . 
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Table 13 



Outcomes of local Action (N=B) 



fin t" f*t\mo c? 

uutcouies 


Number of Events 
TrlpppHnE Outcomes 
(N=10) 


Types of Events 
TriceerinK Outcomes 




Change in practice/policy affecting... 






- specific program system-wide 


2 


SC, RW (P) 


- specific program in one or two schools 


2 


SC 


- some teacners in a specnic program ui 




SF (P), LW 


school (conducted by local participant) 


2 


Inservice attended by. . , 






- 10-20 school-based staff 


1 


SC, SF (P) 


- 20-160 school-based staff 


1 


SC 


Increased expertise (on the job use) by 






participants of MSDE event 






- central office staff 


2 


SC, SF (P) 


- school-based staff 


4 


RW, LF (P), LW 


- other 


2 


SC, RW (P) 


Increased knowledge through exchange of 






information 






- central of fice staff 


2 


se 


- school-based staff 


2 


SF (P), LF (P) 


- other 


2 


se, rw 



SC e state conference SF = state follow-up 
RW ■ regional workshop LF - iocai follow-up 
LW = local workshop (?) a pre-contract 

o 

ERIC 

36 



variables being highly influential. LeveJ of use and quality of impact were 



Sometimes increased when MSDE staff invested effort in working fairly 
intensively with a small number of participants. 

The single strongest factor for implementation was the extent to which 
the MSDE information (content of the inservice event) matched a current local 
priority. 

Summary and Conclusions 
A pilot study focusing bh ten MSDE-sponsored inservice events found that 

• planning varied in effort invested by MSDE and by the extent 

of interagency participation, with the strongest positive 

characteristic beiilg the extent to which the coordinator took 
into account the needs and interests of potential participants 

• other positive characteristics of planning included^ careful 
preparation of outside presenters, and matching content (rele- 
vance, level, amount, credibility) , and process to objectives 
and participant needs 

• negative characteristics of planning included: poor selection 
or preparation of presenters, lack of clarification of the 
"real" audience to be addressed, development of a design with 
insufficient time for participant interaction or with use of 
time not reflecting the relative importance of objectives 
addressed 

• both positive and negative characteristics of planning 
(apparent in the planning process and agenda/design of ah \ 
event) were reflected in participant evaluations and in | 
subsequent actions taken (or not taken) by participants 

• objectives focused primarily on awareness — rationale and 
theory building (51%), followed by skill building — practice/ 
feedback (34%) , then concept building — demonstration/ 
modeling (13%), and application — on-site coaching (2%) 

• activities expanded slightly on objectives 

• objectives and activities differed very little among the 
various kinds of events (statewide conference, follow-up, 
regional or local workshop) i although somewhat more time was 
spent on rationale and theory building for events that were 
new initiatives 

• pre-contracted events included somewhat more participant 
interaction than did others, and the former were more likely 
to achieve the outcomes intended by the coordinators 
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• in genetal* participah rated events as good to excellent on 
such criteria as quality* relevance, use of time etc, , with 
one exception (a state conference) 

• intended usee (of ideas presented) stated by participants at 
the time of a given event focused primarily on on-the-j ob use 
by individual participants, and sharing information informally 
with colleagues 

• ie3s than 50% planned to conduct local inservice (turnkey 
training) 

• over 50% planned to incorporate information into system policy 
or practice 

• actions actually taken (about 'six months after a given event) 
were less than intended use (stated at the time of the event), 
particularly in relation to change in policy/practice 

• outcomes of local action included increased knowledge of 
people other than participants of MSDE events, increased 
expertise by participants , participation in turnkey training 
activities, and changes in practice and policy by teachers, 
schools, and across systems 

• local action was most influenced by: relationship of high 
quality content to a current local priority 

• action was not taken when there was insufficient time or 
participants considered the ideas to be irrelevant 

• the type of event (new initiative vs. follow-up, statewide vs. 
local) made no significant difference to the outcomes, but 
pre-contracting did influence action 

• the type of activities had some influence on outcomes when the 
content was complex or could only be used by people inter- 
acting with each other: in both cases participants needed 
practice and feedback if application was to be successful. 

These findings suggest that MSDE coordinators are most likely to influence 

participants 1 behavior to any great extent if they pre-contract. If local 

priorities in the content already exist, interactive planning to apply all 

four levels of objectives and activities can begin. If doubts exist — 

perhaps about a new content — MSDE coordinators need to involve participants 

in planning an event focusing on rationale and theory building to explore 
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potential for application* Outside presenters should be "coached 11 to 
understand participant needs and to design activities which involve partici- 
pants in productive information exchange. 

While current practice requires statements of objectives (of a given 
event) by coordinators, it would be desirable if statements of purpose were 
also required — relating an event to other activities or MSDE priorities, and 
specifying intended outcomes of the event and of the project or priority to 
which it was related. At present some inservice events seem unrelated to 
larger scale activities and intended outcomes are unclear (to coordinators arid 
participants) . 

Li 

Given the MSDE interest in Bruce Joyce's work, and the findings of this 
study, it might be useful to explore the extent to which Joyce's findings can, 
or should be transferred to state-sponsored inservice. For instance, if most 
participants are administrators who are accountable for carrying out local 
priorities or who may need to "vertically transfer" ideas presented at an MSDE 
event, rationale and theory building may be sufficient if state and local 
priorities match. However, increased emphasis on processes of planned change 
(through demonstration and practice) may be necessary for vertical transfer of 
ideas which are not high local priorities 

Given the variety of purposes served by state-sponsored inservice, various 
outcomes can be expected. However, before high-cost training is undertaken, 
the intended outcomes need to be clearly understood, and trainers/sponsors 
need to negotiate with trainees/policy makers to ensure a common understanding 
arid determine mutually relevant involvement. Large scale staff-development 
requires contingency planning rather than a routinized training design. The 
work of such researchers as Knowles, Joyce, and Showers is valuable, but 
cannot be used as a "rule book" when flexibility may be more appropriate. 



